Reinforcement Learning With Function Approximation for Traffic Signal Control

We propose, for the first time, a reinforcement learning (RL) algorithm with function approximation for traffic signal control. Our algorithm incorporates state-action features and is easily implementable in high-dimensional settings. Prior work, e. g., the work of Abdulhai et al., on the application of RL to traffic signal control requires full-state representations and cannot be implemented, even in moderate-sized road networks, because the computational complexity exponentially grows in the numbers of lanes and junctions. We tackle this problem of the curse of dimensionality by effectively using feature-based state representations that use a broad characterization of the level of congestion as low, medium, or high. One advantage of our algorithm is that, unlike prior work based on RL, it does not require precise information on queue lengths and elapsed times at each lane but instead works with the aforementioned described features. The number of features that our algorithm requires is linear to the number of signaled lanes, thereby leading to several orders of magnitude reduction in the computational complexity. We perform implementations of our algorithm on various settings and show performance comparisons with other algorithms in the literature, including the works of Abdulhai et al. and Cools et al., as well as the fixed-timing and the longest queue algorithms. For comparison, we also develop an RL algorithm that uses full-state representation and incorporates prioritization of traffic, unlike the work of Abdulhai et al. We observe that our algorithm outperforms all the other algorithms on all the road network settings that we consider.

Multi-agent Reinforcement Learning for Traffic Signal Control

Optimal control of traffic lights at junctions or traffic signal control (TSC) is essential for reducing the average delay experienced by the road users amidst the rapid increase in the usage of vehicles. In this paper, we formulate the TSC problem as a discounted cost Markov decision process (MDP) and apply multi-agent reinforcement learning (MARL) algorithms to obtain dynamic TSC policies. We model each traffic signal junction as an independent agent. An agent decides the signal duration of its phases in a round-robin (RR) manner using multi-agent Q-learning with either is an element of-greedy or UCB 3] based exploration strategies. It updates its Q-factors based on the cost feedback signal received from its neighbouring agents. This feedback signal can be easily constructed and is shown to be effective in minimizing the average delay of the vehicles in the network. We show through simulations over VISSIM that our algorithms perform significantly better than both the standard fixed signal timing (FST) algorithm and the saturation balancing (SAT) algorithm 15] over two real road networks.

Optimization of adaptive traffic signal control with transit signal priority at isolated intersections using parallel genetic algorithms

Optimization of adaptive traffic signal timing is one of the most complex problems in traffic control systems. This dissertation presents a new method that applies the parallel genetic algorithm (PGA) to optimize adaptive traffic signal control in the presence of transit signal priority (TSP). The method can optimize the phase plan, cycle length, and green splits at isolated intersections with consideration for the performance of both the transit and the general vehicles. Unlike the simple genetic algorithm (GA), PGA can provide better and faster solutions needed for real-time optimization of adaptive traffic signal control. ^ An important component in the proposed method involves the development of a microscopic delay estimation model that was designed specifically to optimize adaptive traffic signal with TSP. Macroscopic delay models such as the Highway Capacity Manual (HCM) delay model are unable to accurately consider the effect of phase combination and phase sequence in delay calculations. In addition, because the number of phases and the phase sequence of adaptive traffic signal may vary from cycle to cycle, the phase splits cannot be optimized when the phase sequence is also a decision variable. A "flex-phase" concept was introduced in the proposed microscopic delay estimation model to overcome these limitations. ^ The performance of PGA was first evaluated against the simple GA. The results show that PGA achieved both faster convergence and lower delay for both under- or over-saturated traffic conditions. A VISSIM simulation testbed was then developed to evaluate the performance of the proposed PGA-based adaptive traffic signal control with TSP. The simulation results show that the PGA-based optimizer for adaptive TSP outperformed the fully actuated NEMA control in all test cases. The results also show that the PGA-based optimizer was able to produce TSP timing plans that benefit the transit vehicles while minimizing the impact of TSP on the general vehicles. The VISSIM testbed developed in this research provides a powerful tool to design and evaluate different TSP strategies under both actuated and adaptive signal control. ^

MOTION - A New On-line Model for Urban Traffic Signal Control.

This paper presents MOTION, a modular on-line model for urban traffic signal control. It consists of a network and a local level and builds on enhanced traffic state estimation. Special consideration is given to the prioritization of public transit. MOTION provides possibilities for the interaction with integrated urban management systems.

HERMES - High Efficiency Roads with Rerouteing Methods and Traffic Signal Control.

HERMES is one of the projects in the European ATT Programme. The ATT Programme (or DRIVE II as it is frequently referred to) is an application oriented Community Research and Technological Development Programme that has been conceived and implemented with the objective of contributing to the competitiveness of Europe and to its social and economic cohesion. An important means toward this end is the direct collaboration between different European sector actors: road authorities, fleet operators, road user representatives, industry, and research institutions. DRIVE I has already achieved an important step into this direction. DRIVE II aims at providing a framework that encourages even closer cooperation through large scale international pilot projects that will require common functional and technical specifications for the systems to be implemented at least between the partners directly involved in any project. HERMES is one of the so-called "supporting R&D projects" that provides strategies, algorithms and systems for the pilot applications

Optimized strategy for integrated traffic and transit signal control

This paper proposes a unique and innovative approach to integrate transit signal priority control into a traffic adaptive signal control strategy. The proposed strategy was named OSTRAC (Optimized Strategy for integrated TRAffic and TRAnsit signal Control). The cornerstones of OSTRAC include an online microscopic traffic f low prediction model and a Genetic Algorithm (GA) based traffic signal timing module. A sensitivity analysis was conducted to determine the critical GA parameters. The developed traffic f low model demonstrated reliable prediction results through a test. OSTRAC was evaluated by comparing its performance to three other signal control strategies. The evaluation results revealed that OSTRAC efficiently and effectively reduced delay time of general traffic and also transit vehicles.

International comparative field evaluation of a traffic-responsive signal control strategy in three cities

The recently developed network-wide real-time signal control strategy TUC has been implemented in three traffic networks with quite different traffic and control infrastructure characteristics: Chania, Greece (23 junctions); Southampton, UK (53 junctions); and Munich, Germany (25 junctions), where it has been compared to the respective resident real-time signal control strategies TASS, SCOOT and BALANCE. After a short outline of TUC, the paper describes the three application networks; the application, demonstration and evaluation conditions; as well as the comparative evaluation results. The main conclusions drawn from this high-effort inter-European undertaking is that TUC is an easy-to-implement, inter-operable, low-cost real-time signal control strategy whose performance, after very limited fine-tuning, proved to be better or, at least, similar to the ones achieved by long-standing strategies that were in most cases very well fine-tuned over the years in the specific networks.

Comparative field evaluation of the signal control strategy TUC in three traffic networks.

This paper describes how the recently developed network-wide real-time signal control strategy TUC has been implemented in three traffic networks with quite different traffic and control infrastructure characteristics: Chania, Greece (23 junctions); Southampton, U.K. (53 junctions); and Munich, Germany (25 junctions), where it has been compared to the respective resident real-time signal control strategies TASS, SCOOT and BALANCE. After a short outline of TUC, the paper describes the three application networks; the application, demonstration and evaluation conditions; as well as the comparative evaluation results. The main conclusions drawn from this high-effort inter-European undertaking is that TUC is an easy-to-implement, inter-operable, low-cost real-time signal control strategy whose performance, after limited fine-tuning, proved to be better or, at least, similar to the ones achieved by long-standing strategies that were in most cases very well fine-tuned over the years in the specific networks.

Rule-based transit signal priority control method using a real-time transit travel time prediction model

An advanced rule-based Transit Signal Priority (TSP) control method is presented in this paper. An on-line transit travel time prediction model is the key component of the proposed method, which enables the selection of the most appropriate TSP plans for the prevailing traffic and transit condition. The new method also adopts a priority plan re-development feature that enables modifying or even switching the already implemented priority plan to accommodate changes in the traffic conditions. The proposed method utilizes conventional green extension and red truncation strategies and also two new strategies including green truncation and queue clearance. The new method is evaluated against a typical active TSP strategy and also the base case scenario assuming no TSP control in microsimulation. The evaluation results indicate that the proposed method can produce significant benefits in reducing the bus delay time and improving the service regularity with negligible adverse impacts on the non-transit street traffic.

Intelligent traffic light control of isolated intersections using machine learning methods

Traffic congestion is one of the major problems in modern cities. This study applies machine learning methods to determine green times in order to minimize in an isolated intersection. Q-learning and neural networks are applied here to set signal light times and minimize total delays. It is assumed that an intersection behaves in a similar fashion to an intelligent agent learning how to set green times in each cycle based on traffic information. Here, a comparison between Q-learning and neural network is presented. In Q-learning, considering continuous green time requires a large state space, making the learning process practically impossible. In contrast to Q-learning methods, the neural network model can easily set the appropriate green time to fit the traffic demand. The performance of the proposed neural network is compared with two traditional alternatives for controlling traffic lights. Simulation results indicate that the application of the proposed method greatly reduces the total delay in the network compared to the alternative methods.

A review on computational intelligence methods for controlling traffic signal timing

Urban traffic as one of the most important challenges in modern city life needs practically effective and efficient solutions. Artificial intelligence methods have gained popularity for optimal traffic light control. In this paper, a review of most important works in the field of controlling traffic signal timing, in particular studies focusing on Q-learning, neural network, and fuzzy logic system are presented. As per existing literature, the intelligent methods show a higher performance compared to traditional controlling methods. However, a study that compares the performance of different learning methods is not published yet. In this paper, the aforementioned computational intelligence methods and a fixed-time method are implemented to set signals times and minimize total delays for an isolated intersection. These methods are developed and compared on a same platform. The intersection is treated as an intelligent agent that learns to propose an appropriate green time for each phase. The appropriate green time for all the intelligent controllers are estimated based on the received traffic information. A comprehensive comparison is made between the performance of Q-learning, neural network, and fuzzy logic system controller for two different scenarios. The three intelligent learning controllers present close performances with multiple replication orders in two scenarios. On average Q-learning has 66%, neural network 71%, and fuzzy logic has 74% higher performance compared to the fixed-time controller.